Methods of Data Analysis Learning probability distributions

One of the key problems in non-parametric data analysis is to infer good models of probability distributions, assuming we are given as data a finite sample from that distribution. This problem is ill-posed for continuous distributions, even when they are low-dimensional: with finite data, there is no way to distinguish between (or exclude) distributions that are not regularized a priori, for example, by assuming that they are smooth in some sense. The question then becomes about how to formulate the problem of choosing the “best” distribution among the distributions that are “sufficiently smooth” mathematically; intuitively, more data should allow us to consider less smooth distributions. One way of constructing such continuous distribution estimates is by kernel density estimation (KDE), which literally involves smoothing (or convolving) the data with a particular kernel function. Assuming a metric on the state space, the kernel specifies how far away from individual data points the corresponding contributions to estimated distribution extend. Another possibility is to describe the data with a parametric family of distributions that is sufficiently rich so that, in the large sample limit, it can describe an arbitrary smooth distribution. A well-known example from this class are Mixtures of Gaussians. Both of these approaches are reasonably simple to implement, both can be seen as instances of probabilistic (maximum likelihood or Bayesian) inference for their parameters, and both belong to a standard set of unsupervised learning tools for data exploration / modeling. Yet another versatile framework for modeling both continuous and discrete distributions of potentially high dimensionality given a finite sample is the maximum entropy (ME) approach. Here, we are looking for the most random distribution (= maximum entropy) that exactly reproduces a chosen set of statistics which can reliably be estimated from data. The assumption of maximum entropy is a formal version of Occam’s razor: one chooses distributions of a particular form that contain a minimal amount of structure that is nevertheless sufficient to explain selected aspects of observations (constraints). Seen another way, the maximum entropy assumption is just another form of regularization, the intuition being that the most regular smooth distributions are uniform distributions which maximize the entropy by definition. What makes maximum entropy models special are certain uniqueness theorems and links to information-theoretic quantities. Going beyond these standard tools, one can consider unsupervised deep learning models, restricted Boltzmann machines, field-theoretic methods for learning probability distributions and Gaussian Processes (in a supervised setting) that we will consider in the next lecture.